Following @alcuadrado advice, I run the wrapped MPT v4 (version from this PR) against v3.0.0 tests. This helped discover a bug in getRaw and putRaw wrapper implementation. These methods should always use the main DB class instance (stored in _mainDB field) regardless of checkpointing state. The methods were removed in v4 so a polyfill in wrappers was necessary. Along with the methods the tests for them were removed so we didn't notice the problem earlier.

I also added wrappers for some other methods to make the v3 tests pass. They were not wrapped before as they are not used by the packages of interest to us. These steps helped to reduce the number of errors in Buidler EVM tests from 237 down to 12, but there still is some compatibility problem.

Tests output: buidler-core tests on wrapped MPT v4 (level, fixed "raw" methods).txt

CC: @ryanio @holgerd77

I'm glad you found this @msieczko!

Failures 1, 2, 3, 4, 9, 10, 11, and 12 are unrelated. To avoid them, you have to run npm run build before npm run test. This is not very documented, sorry about that.

The other failing tests are related to eth_call, which runs a tx and immediately reverts the state. This is implemented here.

My suspicion is that one of these methods broke:

• StateManager#getStateRoot
• StateManager#setStateRoot
• VM#runTx -- Lower suspicion, as eth_sendTransaction test aren't failing.

@msieczko thanks so much for your debugging and investigation.

I'll add some more "breaking change" release notes based on your insights in this PR, and I will continue to update with any new merges.

The remaining failing tests that @alcuadrado mentioned are in fact 2 tests that fail for the same reason. They both test a simple behaviour and fail at step 3:

1. Deploy a simple storage contract (see it's solidity code below).
2. Send eth_call to get the stored value and confirm that it's all zeros.
3. Send eth_sendTransaction with transaction modifying the stored value.
4. Send eth_call to get the stored value again and check that it has been changed.

pragma solidity 0.5.10;

contract Example {
    event StateModified(uint256 indexed _oldI, uint256 _newI);
    
    
    uint256 public i = 0;
    uint8 public j = 1;
    bytes32 h = "1234567890123456789012345678901234567890123456789012345678901234";
    
    function modifiesState(uint256 _i) payable public {
        emit StateModified(i, _i);
        i = _i;
    }
    
}

What's interesting is that if you omit the 2nd step, both tests pass. This means that eth_call somehow changes the state of the vm or the trie. We're trying to get to the bottom of this at the moment. @ryanio maybe you recognise any changes in the tree that may have caused this or have some idea that would help us debugging?

Ok, I found another deviation in the wrappers which caused these 2 last tests to fail. It turns out that SecureTrie.copy disregards changes introduced during checkpoint whereas CheckpointTrie.copy by default does not. Is this inconsistency intentional?

I've also found that MPT v3 copy methods don't work properly in checkpoint phase, so I assume that dependant packages don't make use of them during checkpoint.

@msieczko thanks for finding these inconsistencies. i'm not sure if they are deliberate or not. maybe we can add some failing test cases and resolve them based on the behavior we should expect to be right?

@ryanio I didn't identify any bug in MPT v4 itself other than this one inconsistency which would cause problems during my integration attempts. I'd say that SecureTrie.copy should behave in the same way as CheckpointTrie.copy (by default) and include checkpoints. However this was not the case in v3 (see this test which is failing) so I guess that's why it's left this way in v4. On the other hand copy method in v3 did not work during checkpoint at all which is demonstrated by this test.

Anyway, packages depending on MPT assume that SecureTrie.copy does not include checkpoints, because this last update to the wrappers fixed the remaining Buidler EVM tests.